Imaging System and Method for Use in Surgical and Interventional Medical Procedures

ABSTRACT

A system and method for displaying images of internal anatomy includes an image processing device configured to provide high resolution images of the surgical field from low resolution scans during the procedure. The image processing device digitally manipulates a previously-obtained high resolution baseline image to produce many representative images based on permutations of movement of the baseline image. During the procedure a representative image is selected having an acceptable degree of correlation to the new low resolution image. The selected representative image and the new image are merged to provide a higher resolution image of the surgical field. The image processing device is also configured to provide interactive movement of the displayed image based on movement of the imaging device, and to permit placement of annotations on the displayed image to facilitate communication between the radiology technician and the surgeon.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and is a continuation-in-part ofU.S. utility patent application Ser. No. 13/253,838, filed on Oct. 5,2011, and incorporates the entire disclosure herein by reference.

BACKGROUND

The present invention contemplates a system and method for altering theway a patient image, such as by X-ray, is viewed and obtained. Moreparticularly, the inventive system and method provides means fordecreasing the overall radiation to which a patient is exposed during asurgical procedure but without significantly sacrificing the quality orresolution of the image obtained.

Many surgical procedures require obtaining an image of the patient'sinternal body structure, such as organs and bones. In some procedures,the surgery is accomplished with the assistance of periodic images ofthe surgical site. Surgery can broadly mean any invasive testing orintervention performed by medical personnel, such as surgeons,interventional radiologists, cardiologists, pain management physicians,and the like. In surgeries and interventions that are in effect guidedby serial imaging, which we will refer to as image guided, frequentpatient images are necessary for the physician's proper placement ofsurgical instruments, be they catheters, needles, instruments orimplants, or performance of certain medical procedures. Fluoroscopy, orfluoro, is one form of intraoperative X-ray and is taken by a fluorounit, also known as a C-arm. The C-arm sends X-ray beams through apatient and takes a picture of the anatomy in that area, such asskeletal and vascular structure. It is, like any picture, atwo-dimensional (2D) image of a three-dimensional (3D) space. However,like any picture taken with a camera, key 3D info may be present in the2D image based on what is in front of what and how big one thing isrelative to another.

A DRR is a digital representation of an X-ray made by taking a CT scanof a patient and simulating taking X-rays from different angles anddistances. The result is that any possible X-ray that can be taken forthat patient can be simulated, which is unique and specific to how thepatient's anatomical features look relative to one another. Because the“scene” is controlled, namely by controlling the virtual location of aC-Arm to the patient and the angle relative to one another, a picturecan be generated that should look like any X-ray taken in the operatingroom (OR).

Many imaging approaches, such as taking fluoro images, involve exposingthe patient to radiation, albeit in small doses. However, in these imageguided procedures, the number of small doses adds up so that the totalradiation exposure can be problematic not only to the patient but alsoto the surgeon or radiologist and others participating in the surgicalprocedure. There are various known ways to decrease the amount ofradiation exposure for a patient/surgeon when an image is taken, butthese approaches come at the cost of decreasing the resolution of theimage being obtained. For example, certain approaches use pulsed imagingas opposed to standard imaging, while other approaches involve manuallyaltering the exposure time or intensity. Narrowing the field of view canpotentially also decrease the area of radiation exposure and itsquantity (as well as alter the amount of radiation “scatter”) but againat the cost of lessening the information available to the surgeon whenmaking a medical decision. Further, often times images taken during asurgical intervention are blocked either by extraneous OR equipment orthe actual instruments/implants used to perform the intervention.Limiting the blocking of the normal anatomy behind those objects wouldhave tangible benefits to the medical community.

There is a need for a an imaging system, that can be used in connectionwith standard medical procedures, that reduces the radiation exposure tothe patient and medical personnel, but without any sacrifice in accuracyand resolution of an X-ray image. There is also a need for an imagingsystem that accounts for instruments and hardware, such as implants,that might otherwise obscure a full view of the surgical site.

SUMMARY

According to one aspect, a system and method is providing for generatinga display of a patient's internal anatomy for use in a surgical orinterventional medical procedure based on a previously acquired highresolution baseline image and a newly acquired low resolution image. Thehigh resolution image may be an image obtained during the procedure or apre-procedure image such as a DRR. The low resolution image may beacquired using a pulse and/or low dose radiation setting. The systemcontemplates an image processing device configured to digitallymanipulate the high resolution baseline image to produce a baselineimage set including representative images of the baseline image at aplurality of permutations of movements of the baseline image in 4D or 6Dspace. The new low resolution image is compared to the baseline imageset to select a representative image having an acceptable degree ofcorrelation with the new image. The image processing device mayimplement algorithms to perform the comparison, such as a principalcomponent analysis or other statistical test. The image processingdevice is further configured to merge the selected representative highresolution image with the new low resolution image to generate a mergedimage to be displayed. The merged image may be further processed toallow alternating between the selected high resolution image and the newlow resolution image, or to adjust the amount that the two images aremerged in the displayed image.

In another feature of the present disclosure, an imaging system mayinclude an image processing device that acts as a viewfinder as theimaging device is moved relative to the patient. In accordance with thisfeature, an image of the surgical field is acquired with the imagingdevice in a first orientation. That acquired image is continuouslydisplayed while the imaging device, patient or patient table is movedfrom the first orientation. This movement is tracked is used the imageprocessing device to move the displayed image in relation to the trackedmovement. With this feature, the display acts as a viewfinder to predicthow a new image would appear if captured at that time by the imagingdevice. This feature can thus be used to determine where the next liveimage of the patient's anatomy will be taken or can be used to assist institching multiple images together to form a larger panoramic view ofthe surgical field. The image processing system may implement softwareadapted to optimize the predicted image and minimize misalignment or offangle appearance of the display. In another aspect, the image processingsystem permits annotation of the displayed image to identify anatomicfeatures or desired image trajectories or alignments.

In a further feature of the disclosed embodiments, a baseline image ofanatomy within a surgical field is acquired in a baseline orientation,and that baseline image is digitally manipulated to produce a baselineimage set including representative images of the baseline image at aplurality of permutations of movements of the baseline image. A newimage of the surgical field in which portions of the anatomy are blockedby objects. This new image is compared to the baseline image set toselect a representative image having an acceptable degree of correlationwith the new image. The image processing system generates a displayedimage showing the surgical field with the blocking objects minimized oreliminated. The system further permits fading the blocked objects in andout of the display.

DESCRIPTION OF THE FIGURES

FIG. 1 is a pictorial view of an image guided surgical setting includingan imaging system and an image processing device, as well as a trackingdevice.

FIG. 2 a is an image of a surgical field acquired using a full dose ofradiation in the imaging system.

FIG. 2 b is an image of the surgical field shown in FIG. 2 a in whichthe image was acquired using a lower dose of radiation.

FIG. 2 c is a merged image of the surgical field with the two imagesshown in FIGS. 2 a-b merged in accordance with one aspect of the presentdisclosure.

FIG. 3 is a flowchart of graphics processing steps undertaken by theimage processing device shown in FIG. 1.

FIG. 4 a is an image of a surgical field including an object blocking aportion of the anatomy.

FIG. 4 b is an image of the surgical field shown in FIG. 4 a with edgeenhancement.

FIGS. 4 c-4 j are images showing the surgical field of FIG. 4 b withdifferent functions applied to determine the anatomic and non-anatomicfeatures in the view.

FIGS. 4 k-4 l are images of a mask generated using a threshold and atable lookup.

FIGS. 4 m-4 n are images of the masks shown in FIGS. 4 k-4 l,respectively, after dilation and erosion.

FIGS. 4 o-4 p are images prepared by applying the masks of FIGS. 4 m-4n, respectively, to the filter image of FIG. 4 b to eliminate thenon-anatomic features from the image.

FIG. 5 a is an image of a surgical field including an object blocking aportion of the anatomy.

FIG. 5 b is an image of the surgical field shown in FIG. 5 a with theimage of FIG. 5 a partially merged with a baseline image to display theblocked anatomy.

FIGS. 6 a-b are baseline and merged images of a surgical field includinga blocking object.

FIGS. 7 a-b are displays of the surgical field adjusted for movement ofthe imaging device or C-arm and providing an indicator of an in-boundsor out-of-bounds position of the imaging device for acquiring a newimage.

FIGS. 8 a-b are displays of the surgical field adjusted for movement ofthe imaging device or C-arm and providing an indicator of when a newimage can be stitched to a previously acquired image.

FIGS. 9 a-b are displays of the surgical field adjusted for movement ofthe imaging device or C-arm and providing an indicator of alignment ofthe imaging device with a desired trajectory for acquiring a new image.

FIG. 10 is a depiction of a display and user interface for the imageprocessing device shown in FIG. 1.

FIG. 11 is a graphical representation of an image alignment processaccording to the present disclosure.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of theinvention, reference will now be made to the embodiments illustrated inthe drawings and described in the following written specification. It isunderstood that no limitation to the scope of the invention is therebyintended. It is further understood that the present invention includesany alterations and modifications to the illustrated embodiments andincludes further applications of the principles of the invention aswould normally occur to one skilled in the art to which this inventionpertains.

A typical imaging system 100 is shown in FIG. 1. The imaging systemincludes a base unit 102 supporting a C-arm imaging device 103. TheC-arm includes a radiation source 104 that is positioned beneath thepatient P and that directs a radiation beam upward to the receiver 105.It is known that the radiation beam emanated from the source 104 isconical so that the field of exposure may be varied by moving the sourcecloser to or away from the patient. The C-arm 103 may be rotated aboutthe patient P in the direction of the arrow 108 for different viewingangles of the surgical site. In some instances, implants or instrumentsT may be situated at the surgical site, necessitating a change inviewing angle for an unobstructed view of the site. Thus, the positionof the receiver relative to the patient, and more particularly relativeto the surgical site of interest, may change during a procedure asneeded by the surgeon or radiologist. Consequently, the receiver 105 mayinclude a tracking target 106 mounted thereto that allows tracking ofthe position of the C-arm using a tracking device 130. For instance, thetracking target 106 may include several infrared emitters spaced aroundthe target, while the tracking device is configured to triangulate theposition of the receiver 105 from the infrared signals emitted by theelement. The base unit 102 includes a control panel 110 through which aradiology technician can control the location of the C-arm, as well asthe radiation exposure. A typical control panel 110 thus permits thetechnician to “shoot a picture” of the surgical site at the surgeon'sdirection, control the radiation dose, and initiate a radiation pulseimage.

The receiver 105 of the C-arm 103 transmits image data to an imageprocessing device 122. The image processing device can include a digitalmemory associated therewith and a processor for executing digital andsoftware instructions. The image processing device may also incorporatea frame grabber that uses frame grabber technology to create a digitalimage for projection as displays 123, 124 on a display device 126. Thedisplays are positioned for interactive viewing by the surgeon duringthe procedure. The two displays may be used to show a images from twoviews, such as lateral and AP, or may show a baseline scan and a currentscan of the surgical site, or a current scan and a “merged” scan basedon a prior baseline scan and a low radiation current scan, as describedherein. An input device 125, such as a keyboard or a touch screen, canallow the surgeon to select and manipulate the on-screen images. It isunderstood that the input device may incorporate an array of keys ortouch screen icons corresponding to the various tasks and featuresimplemented by the image processing device 122. The image processingdevice includes a processor that converts the image data obtained fromthe receiver 105 into a digital format. In some cases the C-arm may beoperating in the cinematic exposure mode and generating many images eachsecond. In these cases, multiple images can be averaged together over ashort time period into a single image to reduce motion artifacts andnoise.

In one aspect of the present invention, the image processing device 122is configured to provide high quality real-time images on the displays123, 124 that are derived from lower detail images obtained using lowerdoses of radiation. By way of example, FIG. 2 a is a “full dose” (FD)x-ray image, while FIG. 2 b is a low dose and/or pulsed (LD) image ofthe same anatomy. It is apparent that the LD image is too “noisy” anddoes not provide enough information about the local anatomy for accurateimage guided surgery. While the FD image provides a crisp view of thesurgical site, the higher radiation dose makes taking multiple FD imagesduring a procedure highly problematic. Using the steps described herein,the surgeon is provided with a current image shown in FIG. 2 c thatsignificantly reduces the noise of the LD image, in some cases by about90%, so that surgeon is provided with a clear real-time image using apulsed or low dose radiation setting. This capability allows fordramatically less radiation exposure during the imaging to verify theposition of instruments and implants during the procedure.

The flowchart of FIG. 3 depicts one embodiment of method according tothe present invention. In a first step 200, a baseline high resolutionFD image is acquired of the surgical site and stored in a memoryassociated with the image processing device. In some cases where theC-arm is moved during the procedure, multiple high resolution images canbe obtained at different locations in the surgical site, and then thesemultiple images “stitched” together to form a composite base image (asdiscussed below). Movement of the C-arm, and more particularly“tracking” the acquired image during these movements, is accounted forin other steps described in more detail herein. For the presentdiscussion it is assumed that the imaging system is relative fixed,meaning that only very limited movement of the C-arm and/or patient arecontemplated, such as might arise in an epidural pain procedure, spinalK-wire placement or stone extraction. The baseline image is projected instep 202 on the display 123 for verification that the surgical site isproperly centered within the image. In some cases, new FD images may beobtained until a suitable baseline image is obtained. In procedures inwhich the C-arm is moved, new baseline images are obtained at the newlocation of the imaging device, as discussed below. If the displayedimage is acceptable as a baseline image, a button may be depressed on auser interface, such as on the display device 126 or interface 125. Inprocedures performed on anatomical regions where a substantial amount ofmotion due to physiological processes (such as respiration) is expected,multiple baseline images may be acquired for the same region overmultiple phases of the cycle. These images may be tagged to temporaldata from other medical instruments, such as an ECG or pulse oximeter.

Once the baseline image is acquired, a baseline image set is generatedin step 204 in which the original baseline image is digitally rotated,translated and resized to create thousands of permutations of theoriginal baseline image. For instance, a typical two dimensional (2D)image of 128×128 pixels may be translated ±15 pixels in the x and ydirections at 1 pixel intervals, rotated ±9° at 3° intervals and scaledfrom 92.5% to 107.5% at 2.5% intervals (4 degrees of freedom, 4D),yielding 47,089 images in the baseline image set. (A three-dimensional(3D) image will imply a 6D solution space due to the addition of twoadditional rotations orthogonal to the x and y axis. An original CTimage data set can be used to form many thousands of DRRs in a similarfashion.) Thus, in this step, the original baseline image spawnsthousands of new image representations as if the original baseline imagewas acquired at each of the different movement permutations. This“solution space” may be stored in a graphics card memory, such as in thegraphics processing unit (GPU) of the image processing device 122, instep 206 or formed as a new image which is then sent to the GPU,depending on the number of images in the solution space and the speed atwhich the GPU can produce those images. With current computing power, ona free standing, medical grade computer, the generation of a baselineimage set having nearly 850,000 images can occur in less than one secondin a GPU because the multiple processors of the GPU can eachsimultaneously process an image.

During the procedure, a new LD image is acquired in step 208, stored inthe memory associated with the image processing device, and projected ondisplay 123. Since the new image is obtained at a lower dose ofradiation it is very noisy. The present invention thus provides stepsfor “merging” the new image with an image from the baseline image set toproduce a clearer image on the second display 124 that conveys moreuseful information to the surgeon. The invention thus contemplates animage recognition or registration step 210 in which the new image iscompared to the images in the baseline image set to find a statisticallymeaningful match. A new “merged” image is generated in step 212 that maybe displayed on display 124 adjacent the view of the original new image.At various times throughout the procedure, a new baseline image may beobtained in step 216 that is used to generate a new baseline image setin step 204.

Step 210 contemplates comparing the current new image to the images inthe baseline image set. Since this step occurs during the surgicalprocedure, time and accuracy are critical. Preferably, the step canobtain an image registration in less than one second so that there is nomeaningful delay between when the image is taken by the C-arm and whenthe merged image is displayed on the device 126. Various algorithms maybe employed that may be dependent on various factors, such as the numberof images in the baseline image set, the size and speed of the computerprocessor or graphics processor performing the algorithm calculations,the time allotted to perform the computations, and the size of theimages being compared (e.g., 128×128 pixels, 1024×1024 pixels, etc). Inone approach, comparisons are made between pixels at predeterminedlocations described above in a grid pattern throughout 4D space. Inanother heuristic approach, pixel comparisons can be concentrated inregions of the images believed to provide a greater likelihood of arelevant match. These regions may be “pre-seeded” based on knowledgefrom a grid or PCA search (defined below), data from a tracking system(such as an optical surgical navigation device), or location data fromthe DICOM file or the equivalent. Alternatively, the user can specifyone or more regions of the image for comparison by marking on thebaseline image the anatomical features considered to be relevant to theprocedure. With this input each pixel in the region can be assigned arelevance score between 0 and 1 which scales the pixel's contribution tothe image similarity function when a new image is compared to thebaseline image. The relevance score may be calibrated to identifyregion(s) to be concentrated on or region(s) to be ignored.

In another approach, a principal component analysis (PCA) is performed,which can allow for comparison to a larger number of larger images inthe allotted amount of time than is permitted with the full resolutiongrid approach. In the PCA approach, a determination is made as to howeach pixel of the image set co-varies with each other. A covariancematrix may be generated using only a small portion of the total solutionset—for instance, a randomly selected 10% of the baseline image set.Each image from the baseline image set is converted to a column vector.In one example, a 70×40 pixel image becomes a 2800×1 vector. Thesecolumn vectors are normalized to a mean of 0 and a variance of 1 andcombined into a larger matrix. The covariance matrix is determined fromthis larger matrix and the largest eigenvectors are selected. For thisparticular example, it has been found that 30 PCA vectors can explainabout 80% of the variance of the respective images. Thus, each 2800×1image vector can be multiplied by a 2800×30 PCA vector to yield a 1×30vector. The same steps are applied to the new image—the new image isconverted to a 2800×1 image vector and multiplication with the 2800×30PCA vector produces a 1×30 vector corresponding to the new image. Thesolution set (baseline image) vectors and the new image vector arenormalized and the dot product of the new image vector to each vector inthe solution space is calculated. The solution space baseline imagevector that yields the largest dot product (i.e., closest to 1) isdetermined to be the closest image to the new image. It is understoodthat the present example may be altered with different image sizesand/or different principal components used for the analysis. It isfurther understood that other known techniques may be implemented thatmay utilize eigenvectors, singular value determination, mean squarederror, mean absolute error, and edge detection, for instance. It isfurther contemplated that various image recognition approaches can beapplied to selected regions of the images or that various statisticalmeasures may be applied to find matches falling within a suitableconfidence threshold. A confidence or correlation value may be assignedthat quantifies the degree of correlation between the new image and theselected baseline image, or selected ones of the baseline image set, andthis confidence value may be displayed for the surgeon's review. Thesurgeon can decide whether the confidence value is acceptable for theparticular display and whether another image should be acquired.

In the image guided surgical procedures, tools, implants and instrumentswill inevitably appear in the image field. These objects are typicallyradiodense and consequently block the relevant patient anatomy fromview. The new image obtained in step 210 will thus include an artifactof the tool T that will not correlate to any of the baseline image set.The presence of the tool in the image thus ensures that the comparisontechniques described above will not produce a high degree ofregistration between the new image and any of the baseline image set.Nevertheless, if the end result of each of the above procedures is toseek out the highest degree of correlation, which is statisticallyrelevant or which exceeds a certain threshold, the image registrationmay be conducted with the entire new image, tool artifact and all.

Alternatively, the image registration steps may be modified to accountfor the tool artifacts on the new image. In one approach, the new imagemay be evaluated to determine the number of image pixels that are“blocked” by the tool. This evaluation can involve comparing a grayscalevalue for each pixel to a threshold and excluding pixels that falloutside that threshold. For instance, if the pixel grayscale values varyfrom 0 (completely blocked) to 10 (completely transparent), a thresholdof 3 may be applied to eliminate certain pixels from evaluation.Additionally, when location data is available for various tracked tools,algorithmically areas that are blocked can be mathematically avoided.

In another approach, the image recognition or registration step 210 mayinclude steps to measure the similarity of the LD image to a transformedversion of the baseline image (i.e., a baseline image that has beentransformed to account for movement of the C-arm, as described belowrelative to FIG. 11) or of the patient. In an image-guided surgicalprocedure, the C-arm system acquires multiple X-ray images of the sameanatomy. Over the course of this series of images the system may move insmall increments and surgical tools may be added or removed from thefield of view, even though the anatomical features may remain relativelystable. The approach described below takes advantage of this consistencyin the anatomical features by using the anatomical features present inone image to fill in the missing details in another later image. Thisapproach further allows the transfer of the high quality of a full doseimage to subsequent low dose images.

In the present approach, a similarity function in the form of a scalarfunction of the images is used to determine the registration between acurrent LD image and a baseline image. To determine this registration itis first necessary to determine the incremental motion that has occurredbetween images. This motion can be described by four numberscorresponding to four degrees of freedom—scale, rotation and verticaland horizontal translation. For a given pair of images to be comparedknowledge of these four numbers allows one of the images to bemanipulated so that the same anatomical features appear in the samelocation between both images. The scalar function is a measure of thisregistration and may be obtained using a correlation coefficient, dotproduct or mean square error. By way of example, the dot product scalarfunction corresponds to the sum of the products of the intensity valuesat each pixel pair in the two images. For example, the intensity valuesfor the pixel located at 1234, 1234 in each of the LD and baselineimages are multiplied. A similar calculation is made for every otherpixel location and all of those multiplied values are added for thescalar function. It can be appreciated that when two images are in exactregistration this dot product will have the maximum possible magnitude.In other words, when the best combination is found, the correspondingdot product it typically higher than the others, which may be reportedas the Z score (i.e., number of standard deviations above the mean). A Zscore greater than 7.5 represents a 99.9999999% certainty that theregistration was not found by chance. It should be borne in mind thatthe registration being sought using this dot product is between abaseline image of a patient's anatomy and a real-time low dose image ofthat same anatomy taken at a later time after the viewing field andimaging equipment may have moved or non-anatomical objects introducedinto the viewing field.

This approach is particularly suited to performance using a parallelcomputing architecture such as the GPU which consists of multipleprocessors capable of performing the same computation in parallel. Eachprocessor of the GPU may thus be used to compute the similarity functionof the LD image and one transformed version of the baseline image. Inthis way, multiple transformed versions of the baseline image can becompared to the LD image simultaneously. The transformed baseline imagescan be generated in advance when the baseline is acquired and thenstored in GPU memory. Alternatively, a single baseline image can bestored and transformed on the fly during the comparison by reading fromtransformed coordinates with texture fetching. In situations in whichthe number of processors of the GPU greatly exceeds the number oftransformations to be considered, the baseline image and the LD imagecan be broken into different sections and the similarity functions foreach section can be computed on different processors and thensubsequently merged.

To further accelerate the determination of the best transformation toalign two images, the similarity functions can first be computed withdown-sampled images that contain fewer pixels. This down-sampling can beperformed in advance by averaging together groups of neighboring pixels.The similarity functions for many transformations over a broad range ofpossible motions can be computed for the down-sampled images first. Oncethe best transformation from this set is determined that transformationcan be used as the center for a finer grid of possible transformationsapplied to images with more pixels. In this way, multiple steps are usedto determine the best transformation with high precision whileconsidering a wide range of possible transformations in a short amountof time.

In order to reduce the bias to the similarity function caused bydifferences in the overall intensity levels in the different images, andto preferentially align anatomical features in the images that are ofinterest to the user, the images can be filtered before the similarityfunction is computed. Such filters will ideally suppress the very highspatial frequency noise associated with low dose images, while alsosuppressing the low spatial frequency information associated with large,flat regions that lack important anatomical details. This imagefiltration can be accomplished with convolution, multiplication in theFourier domain or Butterworth filters, for example. It is thuscontemplated that both the LD image and the baseline image(s) will befiltered accordingly prior to generating the similarity function.

As previously explained, non-anatomical features may be present in theimage, such as surgical tools, in which case modifications to thesimilarity function computation process may be necessary to ensure thatonly anatomical features are used to determine the alignment between LDand baseline images. A mask image can be generated that identifieswhether or not a pixel is part of an anatomical feature. In one aspect,an anatomical pixel may be assigned a value of 1 while a non-anatomicalpixel is assigned a value of 0. This assignment of values allows boththe baseline image and the LD image to be multiplied by thecorresponding mask images before the similarity function is computed asdescribed above In other words, the mask image can eliminate thenon-anatomical pixels to avoid any impact on the similarity functioncalculations.

To determine whether or not a pixel is anatomical, a variety offunctions can be calculated in the neighborhood around each pixel. Thesefunctions of the neighborhood may include the standard deviation, themagnitude of the gradient, and/or the corresponding values of the pixelin the original grayscale image and in the filtered image. The“neighborhood” around a pixel includes a pre-determined number ofadjacent pixels, such as a 5×5 or a 3×3 grid. Additionally, thesefunctions can be compounded, for example, by finding the standarddeviation of the neighborhood of the standard deviations, or bycomputing a quadratic function of the standard deviation and themagnitude of the gradient. One example of a suitable function of theneighborhood is the use of edge detection techniques to distinguishbetween bone and metallic instruments. Metal presents a “sharper” edgethan bone and this difference can be determined using standard deviationor gradient calculations in the neighborhood of an “edge” pixel. Theneighborhood functions may thus determine whether a pixel is anatomic ornon-anatomic based on this edge detection approach and assign a value of1 or 0 as appropriate to the pixel.

Once a set of values has been computed for the particular pixel, thevalues can be compared against thresholds determined from measurementsof previously-acquired images and a binary value can be assigned to thepixel based on the number of thresholds that are exceeded.Alternatively, a fractional value between 0 and 1 may be assigned to thepixel, reflecting a degree of certainty about the identity of the pixelas part of an anatomic or non-anatomic feature. These steps can beaccelerated with a GPU by assigning the computations at one pixel in theimage to one processor on the GPU, thereby enabling values for multiplepixels to be computed simultaneously. The masks can be manipulated tofill in and expand regions that correspond to non-anatomical featuresusing combinations of morphological image operations such as erosion anddilation.

An example of the steps of this approach is illustrated in the images ofFIGS. 4 a-4 p. In FIG. 4 a, an image of a surgical site includesanatomic features (the patient's skull) and non-anatomic features (suchas a clamp). The image of FIG. 4 a is filtered for edge enhancement toproduce the filtered image of FIG. 4 b. It can be appreciated that thisimage is represented by thousands of pixels in a conventional manner,with the intensity value of each pixel modified according to the edgeenhancement attributes of the filter. In this example, the filter is aButterworth filter. This filtered image is then subject to eightdifferent techniques for generating a mask corresponding to thenon-anatomic features. Thus, the neighborhood functions described above(namely, standard deviation, gradient and compounded functions thereof)are applied to the filtered image FIG. 4 b to produce different imagesFIGS. 4 c-4 j. Each of these images is stored as a baseline image forcomparison to and registration with a live LD image.

Thus, each image of FIGS. 4 c-4 j is used to generate a mask. Asexplained above, the mask generation process may be by comparison of thepixel intensities to a threshold value or by a lookup table in whichintensity values corresponding to known non-anatomic features iscompared to the pixel intensity. The masks generated by the thresholdand lookup table techniques for one of the neighborhood function imagesis shown in FIGS. 4 k-4 l. The masks can then be manipulated to fill inand expand regions that correspond to the non-anatomical features, asrepresented in the images of FIGS. 4 m-4 n. The resulting mask is thenapplied to the filtered image of FIG. 4 b to produce the “final”baseline images of FIGS. 4 o-4 p that will be compared to the live LDimage. As explained above, each of these calculations and pixelevaluations can be performed in the individual processors of the GPU sothat all of these images can be generated in an extremely short time.Moreover, each of these masked baseline images can be transformed toaccount for movement of the surgical field or imaging device andcompared to the live LD image to find the baseline image that yields thehighest Z score corresponding to the best alignment between baseline andLD images. This selected baseline image is then used in manner explainedbelow.

Once the image registration is complete, the new image may be displayedwith the selected image from the baseline image set in different ways.In one approach, the two images are merged, as illustrated in FIGS. 5 a,b. The original new image is shown in FIG. 5 a with the instrument Tplainly visible and blocking the underlying anatomy. A partially mergedimage generated in step 212 (FIG. 3) is shown in FIG. 5 b in which theinstrument T is still visible but substantially mitigated and theunderlying anatomy is visible. The two images may be merged by combiningthe digital representation of the images in a conventional manner, suchas by adding or averaging pixel data for the two images. In oneembodiment, the surgeon may identify one or more specific regions ofinterest in the displayed image, such as through the user interface 125,and the merging operation can be configured to utilize the baselineimage data for the display outside the region of interest and conductthe merging operation for the display within the region of interest. Theuser interface 125 may be provided with a “slider” that controls theamount the baseline image versus the new image that is displayed in themerged image. In another approach, the surgeon may alternate between thecorrelated baseline image and the new image or merged image, as shown inFIGS. 6 a, b. The image in FIG. 6 a is the image from the baseline imageset found to have the highest degree of correlation to the new image.The image in FIG. 6 b is the new image obtained. The surgeon mayalternate between these views to get a clearer view of the underlyinganatomy and a view of the current field with the instrumentation T,which in effect by alternating images digitally removes the instrumentfrom the field of view, clarifying its location relative to the anatomyblocked by it.

In another approach, a logarithmic subtraction can be performed betweenthe baseline image and the new image to identify the differences betweenthe two images. The resulting difference image (which may contain toolsor injected contrast agent that are of interest to the surgeon) can bedisplayed separately, overlaid in color or added to the baseline image,the new image or the merged image so that the features of interestappear more obvious. This may require the image intensity values to bescaled prior to subtraction to account for variations in the C-armexposure settings. Digital image processing operations such as erosionand dilation can be used to remove features in the difference image thatcorrespond to image noise rather than physical objects. The approach maybe used to enhance the image differences, as described, or to remove thedifference image from the merged image. In other words, the differenceimage may be used as a tool for exclusion or inclusion of the differenceimage in the baseline, new or merged images.

As indicated above, the present invention also contemplates a surgicalnavigation procedure in which the imaging device or C-arm 103 is moved.Thus, the present invention contemplates tracking the position of theC-arm rather than tracking the position of the surgical instruments andimplants as in traditional surgical navigation techniques, usingcommercially available tracking devices or the DICOM information fromthe imaging device. Tracking the C-arm requires a degree of accuracythat is much less than the accuracy required to track the instrumentsand implants. In this embodiment, the image processing device 122receives tracking information from the tracking device 130. The objectof this aspect of the invention is to ensure that the surgeon sees animage that is consistent with the actual surgical site regardless of theorientation of the C-arm relative to the patient.

Tracking the position of the C-arm can account for “drift”, which is agradual misalignment of the physical space and the imaging (or virtual)space. This “drift” can occur because of subtle patient movements,inadvertent contact with the table or imaging device and even gravity.This misalignment is often visually imperceptible, but can generatenoticeable shifts in the image viewed by the surgeon. These shifts canbe problematic when the surgical navigation procedure is being performed(and a physician is relying on the information obtained from thisdevice) or when alignment of new to baseline images is required toimprove image clarity. The use of image processing eliminates theinevitable misalignment of baseline and new images. The image processingdevice 122 further may incorporate a calibration mode in which thecurrent image of the anatomy is compared to the predicted image. Thedifference between the predicted and actual movement of the image can beaccounted for by an inaccurate knowledge of the “center of mass” or COM,described below, and drift. Once a few images are obtained and the COMis accurately established, recalibration of the system can occurautomatically with each successive image taken and thereby eliminatingthe impact of drift.

The image processing device 122 may operate in a “tracking mode” inwhich the movement of the C-arm is monitored and the currently displayedimage is moved accordingly. The currently displayed image may be themost recent baseline image, a new LD image or a merged image generatedas described above. This image remains on one of the displays 123, 124until a new picture is taken by the imaging device 100. This image isshifted on the display to match the movement of the C-arm using theposition data acquired by the tracking device 130. A tracking circle 240may be shown on the display, as depicted in FIGS. 7 a, 6 b. The trackingcircle identifies an “in bounds” location for the image. When thetracking circle appears in red, the image that would be obtained withthe current C-arm position would be “out of bounds” in relation to abaseline image position, as shown in FIG. 7 a. As the C-arm is moved bythe radiology technician the representative image on the display ismoved. When the image moves “in bounds”, as shown in FIG. 7 b, thetracking circle 240 turns green so that the technician has an immediateindication that the C-arm is now in a proper position for obtaining anew image. The tracking circle may be used by the technician to guidethe movements of the C-arm during the surgical procedure. The trackingcircle may also be used to assist the technician in preparing a baselinestitched image. Thus, an image position that is not properly aligned forstitching to another image, as depicted in FIG. 8 a, will have a redtracking circle 240, while a properly aligned image position, as shownin FIG. 8 b, will have a green tracking circle. The technician can thenacquire the image to form part of the baseline stitched image.

The present invention contemplates a feature that enhances thecommunication between the surgeon and the radiology technician. Duringthe course of a procedure the surgeon may request images at particularlocations or orientations. One example is what is known as a “Fergusonview” in spinal procedures in which an AP oriented C-arm is canted toalign directly over a vertebral end plate with the end plate oriented“flat” or essentially parallel with the beam axis of the C-arm.Obtaining a Ferguson view requires rotating the C-arm or the patienttable while obtaining multiple AP views of the spine, which iscumbersome and inaccurate using current techniques, requiring a numberof fluoroscopic images to be performed to find the one best aligned tothe endplate. The present invention allows the surgeon to overlay a gridonto a single image or stitched image and provide labels for anatomicfeatures that can then be used by the technician to orient the C-arm.Thus, as shown in FIG. 9 a, the image processing device 122 isconfigured to allow the surgeon to place a grid 245 within the trackingcircle 240 overlaid onto a Lateral image. The surgeon may also locatelabels 250 identifying anatomic structure, in this case spinalvertebrae. In this particular example, the goal is to align the L2-L3disc space with the center grid line 246. To assist the technician, atrajectory arrow 255 is overlaid onto the image to indicate thetrajectory of an image acquired with the C-arm in the current position.As the C-arm moves, changing orientation off of pure AP, the imageprocessing device evaluates the C-arm position data obtained from thetracking device 230 to determine the new orientation for trajectoryarrow 255. The trajectory arrow thus moves with the C-arm so that whenit is aligned with the center grid line 246, as shown in FIG. 9 b, thetechnician can shoot the image knowing that the C-arm is properlyaligned to obtain a Ferguson view along the L3 endplate. Thus,monitoring the lateral view until it is rotated and centered along thecenter grid line allows the radiology technician to find the AP Fergusonangle without guessing and taking a number of incorrect images.

The image processing device may be further configured to show thelateral and AP views simultaneously on respective displays 123 and 124,as depicted in FIG. 10. Either or both views may incorporate the grid,labels and trajectory arrows. This same lateral view may appear on thecontrol panel 110 for the imaging system 100 for viewing by thetechnician. As the C-arm is moved to align the trajectory arrow with thecenter grid line, as described above, both the lateral and AP images aremoved accordingly so that the surgeon has an immediate perception ofwhat the new image will look like. Again, once the technician properlyorients the C-arm, as indicated by alignment of the trajectory arrowwith the center grid line, a new AP image is acquired. As shown in FIG.10, a view may include multiple trajectory arrows, each aligned with aparticular disc space. For instance, the uppermost trajectory arrow isaligned with the L1-L2 disc space, while the lowermost arrow is alignedwith the L5-S1 disc space. In multiple level procedures the surgeon mayrequire a Ferguson view of different levels, which can be easilyobtained by requesting the technician to align the C-arm with aparticular trajectory arrow.

In another feature, a radiodense asymmetric shape can be placed in aknown location on the C-arm detector. This creates the ability to linkthe coordinate frame of the C-arm to the arbitrary orientation of theC-arm's image coordinate frame. As the C-arm's display may be modifiedto generate an image having any rotation or mirroring, detecting thisshape radically simplifies the process of image comparison and imagestitching. Thus, as shown in FIG. 11, the baseline image B includes theindicia “K” at the 9 o'clock position of the image. The new image N isobtained in which the indicia has been rotated by the physician ortechnologist away from the default orientation. Comparing this new imageto the baseline image set is unlikely to produce any registrationbetween images due to this angular offset. In one embodiment, the imageprocessing device detects the actual rotation of the C-arm from thebaseline orientation while in another embodiment the image processingdevice uses image recognition software to locate the “K” indicia in thenew image and determine the angular offset from the default position.This angular offset is used to alter the rotation and/or mirror imagethe baseline image set. The baseline image selected in the imageregistration step 210 is maintained in its transformed orientation to bemerged with the newly acquired image. This transformation can includerotation and mirror-imaging, to eliminate the display effect that ispresent on a C-arm.

In another aspect, it is known that as the C-arm radiation source 104moves closer to the table, the size of the image captured by thereceiver 105 becomes larger; moving the receiver closer to the tableresults in a decrease in image size. Whereas the amount that the imagescales with movements towards and away from the body can be easilydetermined, if the C-arm is translated along the table, the image willshift, with the magnitude of that change depending upon the proximity ofthe “center of mass” (COM) of the patient to the radiation source.Although the imaged anatomy is of 3D structures, with a high degree ofaccuracy, mathematically we can represent this anatomy as a 2D pictureof the 3D anatomy placed at the COM of the structures. Then, forinstance, when the COM is close to the radiation source, small movementswill cause the resulting image to shift greatly. Until the COM isdetermined, though, the calculated amount that the objects on the screenshift will be proportional to but not equal to their actual movement.The difference is used to calculate the actual location of the COM. TheCOM is adjusted based on the amount that those differ, moving it awayfrom the radiation source when the image shifted too much, and theopposite if the image shifts too little. The COM is initially assumed tobe centered on the table to which the reference arc of the trackingdevice is attached. The true location of the COM is fairly accuratelydetermined using the initial two or three images taken during initialset-up of the imaging system, and reconfirmed/adjusted with each newimage taken. Once the COM is determined in global space, the movement ofthe C-arm relative to the COM can be calculated and applied to translatethe baseline image set accordingly for image registration.

The image processing device 122 may also be configured to allow thesurgeon to introduce other tracked elements into an image, to help guidethe surgeon during the procedure. A closed-loop feedback approach allowsthe surgeon to confirm that the location of this perceived trackedelement and the image taken of that element correspond. Specifically,the live x-ray and the determined position from the surgical navigationsystem are compared. In the same fashion that knowledge of the baselineimage, through image recognition, can be used to track the patient'sanatomy even if blocked by radiodense objects, knowledge of theradiodense objects, when the image taken is compared to their trackedlocation, can be used to confirm their tracking When both theinstrument/implant and the C-arm are tracked, the location of theanatomy relative to the imaging source and the location of the equipmentrelative to the imaging source are known. This information can thus beused to quickly and interactively ascertain the location of theequipment or hardware relative to the anatomy. This feature can, by wayof example, have particular applicability to following the path of acatheter in an angio procedure, for instance. In a typical angioprocedure, a cine, or continuous fluoro, is used to follow the travel ofthe catheter along a vessel. The present invention allows intersplicingpreviously generated images of the anatomy with the virtual depiction ofthe catheter with live fluoro shots of the anatomy and actual catheter.Thus, rather than taking 15 fluoro shots per second for a typical cineprocedure, the present invention allows the radiology technician to takeonly one shot per second to effectively and accurately track thecatheter as it travels along the vessel. The previously generated imagesare spliced in to account for the fluoro shots that are not taken. Thevirtual representations can be verified to the live shot when taken andrecalibrated if necessary.

In certain procedures it is possible to fix the position of the vascularanatomy to larger features, such as nearby bones. This can beaccomplished using DRRs from prior CT angiograms (CTA) or from actualangiograms taken in the course of the procedure. Either, approach may beused as a means to link angiograms back to bony anatomy and vice versa.To describe in greater detail, the same CTA may be used to producedifferent DRRs, such as DRRs highlighting just the bony anatomy andanother in a matched set that includes the vascular anatomy along withthe bones. A baseline fluoro image taken of the patient's bony anatomycan then be compared with the bone DRRs to determine the best match.Instead of displaying the result using bone only DRR, the matched DRRthat includes the vascular anatomy can be used to merge with the newimage. In this approach, the bones help to place the radiographicposition of the catheter to its location within the vascular anatomy.Since it is not necessary to continually image the vessel itself, as thepicture of this structure can be overlaid onto the bone only imageobtained, the use of contrast dye can be limited versus prior proceduresin which the contrast dye is necessary to constantly see the vessels.

Following are examples of specific procedures utilizing the features ofthe image processing device discussed above. These are just a fewexamples as to how the software can be manipulated using differentcombinations of baseline image types, display options, and radiationdosing and not meant to be an exhaustive list.

Pulsed New Image/Alternated with/Baseline of FD Fluoro or PreoperativeX-ray

A pulsed image is taken and compared with a previously obtained baselineimage set containing higher resolution non-pulsed image(s) taken priorto the surgical procedure. Registration between the current image andone of the baseline solution set provides a baseline image reflectingthe current position and view of the anatomy. The new image isalternately displayed or overlaid with the registered baseline image,showing the current information overlaid and alternating with the lessobscured or clearer image.

Pulsed New Image/Alternated with/Baseline Derived from DRR

A pulsed image is taken and compared with a previously obtained solutionset of baseline images, containing higher resolution DRR obtained from aCT scan. The DRR image can be limited to just show the bony anatomy, asopposed to the other obscuring information that frequently “cloud” afilm taken in the OR (e.g.—bovie cords, EKG leads, etc.) as well asobjects that obscure bony clarity (e.g.—bowel gas, organs, etc.). Aswith the above example, the new image that is registered with one of theprior DRR images, and these images are alternated or overlaid on thedisplay 123, 124.

Pulsed New Image/Merged instead of Alternated

All of the techniques described above can be applied and instead ofalternating the new and registered baseline images, the prior andcurrent image are merged. By performing a weighted average or similarmerging technique, a single image can be obtained which shows both thecurrent information (e.g.—placement of instruments, implants, catheters,etc.) in reference to the anatomy, merged with a higher resolutionpicture of the anatomy. In one example, multiple views of the merger ofthe two images can be provided, ranging from 100% pulsed image to 100%DRR image. A slide button on the user interface 125 allows the surgeonto adjust this merger range as desired.

New Image is a Small Segment of a Larger Baseline Image Set

The imaging taken at any given time contains limited information, a partof the whole body part. Collimation, for example, lowers the overalltissue radiation exposure and lowers the radiation scatter towardsphysicians but at the cost of limiting the field of view of the imageobtained. Showing the actual last projected image within the context ofa larger image (e.g.—obtained prior, preoperatively or intraoperatively,or derived from CTs)—merged or alternated in the correction location—cansupplement the information about the smaller image area to allow forincorporation into reference to the larger body structure(s). The sameimage registration techniques are applied as described above, exceptthat the registration is applied to a smaller field within the baselineimages (stitched or not) corresponding to the area of view in the newimage.

Same as Above, Located at Junctional or Blocked Areas

Not infrequently, especially in areas that have different overalldensities (e.g.—chest vs. adjacent abdomen, head/neck/cervical spine vs.upper thorax), the area of an x-ray that can be clearly visualized isonly part of the actual image obtained. This can be frustrating to thephysician when it limits the ability to place the narrow view into thelarger context of the body or when the area that needs to be evaluatedis in the obscured part of the image. By stitching together multipleimages, each taken in a localized ideal environment, a larger image canbe obtained. Further, the current image can be added into the largercontext (as described above) to fill in the part of the image clouded byits relative location.

Unblocking the Hidden Anatomy or Mitigating its Local Effects

As described above, the image processing device performs the imageregistration steps between the current new image and a baseline imageset that, in effect, limits the misinformation imparted by noise, be itin the form of x-ray scatter or small blocking objects (e.g.—cords,etc.) or even larger objects (e.g.—tools, instrumentation, etc.). Inmany cases, it is that part of the anatomic image that is being blockedby a tool or instrument that is of upmost importance to the surgerybeing performed. By eliminating the blocking objects from the image thesurgery becomes safer and more efficacious and the physician becomesempowered to continue with improved knowledge. Using an image that istaken prior to the noise being added (e.g.—old films, baseline single FDimages, stitched together fluoro shots taken prior to surgery, etc.) oridealized (e.g.—DRRs generated from CT data), displaying that prior“clean” image, either merged or alternated with the current image, willmake those objects disappear from the image or become shadows ratherthan dense objects. If these are tracked objects, then the blocked areacan be further deemphasized or the information from it can be eliminatedas the mathematical comparison is being performed, further improving thespeed and accuracy of the comparison.

The image processing device configured as described herein providesthree general features that (1) reduce the amount of radiation exposurerequired for acceptable live images, (2) provide images to the surgeonthat can facilitate the surgical procedure, and (3) improve thecommunication between the radiology technician and the surgeon. Withrespect to the aspect of reducing the radiation exposure, the presentinvention permits low dose images to be taken throughout the surgicalprocedure and fills in the gaps created by “noise” in the current imageto produce a composite or merged image of the current field of view withthe detail of a full dose image. In practice this allows for highlyusable, high quality images of the patient's anatomy generated with anorder of magnitude reduction in radiation exposure than standard FDimaging using unmodified features present on all common, commerciallyavailable C-arms. The techniques for image registration described hereincan be implemented in a graphic processing unit and can occur in asecond or so to be truly interactive; when required such as in CINEmode, image registration can occur multiple times per second. A userinterface allows the surgeon to determine the level of confidencerequired for acquiring registered image and gives the surgeon options onthe nature of the display, ranging from side-by-side views to fadein/out merged views.

With respect to the feature of providing images to the surgeon thatfacilitate the surgical procedure, several digital imaging techniquescan be used to improve the user's experience. One example is an imagetracking feature that can be used to maintain the image displayed to thesurgeon in an essentially a “stationary” position regardless of anyposition changes that may occur between image captures. In accordancewith this feature, the baseline image can be fixed in space and newimages adjust to it rather than the converse. When successive images aretaken during a step in a procedure each new image can be stabilizedrelative to the prior images so that the particular object of interest(e.g.—anatomy or instrument) is kept stationary in successive views. Forexample, as sequential images are taken as a bone screw is introducedinto a body part, the body part remains stationary on the display screenso that the actual progress of the screw can be directly observed.

In another aspect of this feature, the current image including blockingobjects can be compared to earlier images without any blocking objects.In the registration process, the image processing device can generate amerged image between new image and baseline image that deemphasizes theblocking nature of the object from the displayed image. The userinterface also provides the physician with the capability to fade theblocking object in and out of the displayed view.

In other embodiments in which the object itself is being tracked, avirtual version of the blocking object can be added back to thedisplayed image. The image processing device can obtain position datafrom a tracking device following the position of the blocking object anduse that position data to determine the proper location and orientationof the virtual object in the displayed image. The virtual object may beapplied to a baseline image to be compared with a new current image toserve as a check step—if the new image matches the generated image (bothtool and anatomy) within a given tolerance then the surgery can proceed.If the match is poor, the surgery can be stopped (in the case ofautomated surgery) and/or recalibration can take place. This allows fora closed-loop feedback feature to facilitate the safety of automation ofmedical intervention.

For certain procedures, such as a pseudo-angio procedure, projecting thevessels from a baseline image onto current image can allow a physicianto watch a tool (e.g.—micro-catheter, stent, etc.) as it travels throughthe vasculature while using much less contrast medium load. The adjacentbony anatomy serves as the “anchor” for the vessels—the bone isessentially tracked, through the image registration process, and thevessel is assumed to stay adjacent to this structure. In other words,when the anatomy moves between successive images, the new image isregistered to a different one of the baseline image set that correspondsto the new position of the “background” anatomy. The vessels from adifferent but already linked baseline image containing the vascularstructures can then be overlaid or merged with the displayed image whichlacks contrast. If necessary or desired, intermittent angios can betaken to confirm. When combined with a tracked catheter, a workingknowledge of the location of the instrument can be included into theimages. A cine (continuous movie loop of fluoro shots commonly used whenan angiogram is obtained) can be created in which generated images areinterspliced into the cine images, allowing for many fewer x-rays to beobtained while an angiogram is being performed or a catheter is beingplaced. Ultimately, once images have been linked to the originalbaseline image, any of these may be used to merge into a current image,producing a means to monitor movement of implants, the formation ofconstructs, the placement of stents, etc.

In the third feature—improving communication—the image processing devicedescribed herein allows the surgeon to annotate an image in a mannerthat can help guide the technician in the positioning of the C-arm as tohow and where to take a new picture. Thus, the user interface 125 of theimage processing device 122 provides a vehicle for the surgeon to add agrid to the displayed image, label anatomic structures and identifytrajectories for alignment of the imaging device. As the technicianmoves the imaging device or C-arm, the displayed image is moved. Thisfeature allows the radiology tech to center the anatomy that is desiredto be imaged in the center of the screen, at the desired orientation,without taking multiple images each time the C-arm is brought back inthe field to obtain this. This feature provides a view finder for theC-arm, a feature lacking currently. The technician can activate theC-arm to take a new image with a view tailored to meet the surgeon'sexpressed need.

In addition, linking the movements of the C-arm to the images takenusing DICOM data or a surgical navigation backbone, for example, helpsto move the displayed image as the C-arm is moved in preparation for asubsequent image acquisition. “In bound” and “out of bounds” indicatorscan provide an immediate indication to the technician whether a currentmovement of the C-arm would result in an image that cannot be correlatedor registered with any baseline image, or that cannot be stitchedtogether with other images to form a composite field of view. The imageprocessing device thus provides image displays that allow the surgeonand technician to visualize the effect of a proposed change in locationand trajectory of the c-arm. Moreover, the image processing device mayhelp the physician, for instance, alter the position of the table or theangle of the C-arm so that the anatomy is aligned properly (such asparallel or perpendicular to the surgical table). The image processingdevice can also determine the center of mass (COM) of the exact centerof an x-rayed object using two or more x-ray shots from two or moredifferent gantry angles/positions, and then use this COM information toimprove the linking of the physical space (in millimeters) to thedisplayed imaging space (in pixels).

The image recognition component disclosed herein can overcome the lackof knowledge of the location of the next image to be taken, whichprovides a number of benefits. Knowing roughly where the new image iscentered relative to the baseline can limit the need to scan a largerarea of the imaging space and, therefore, significantly increase thespeed of image recognition software. Greater amounts of radiationreduction (and therefore noise) can be tolerated, as there exists aninternal check on the image recognition. Multiple features that aremanual in the system designed without surgical navigation, such asbaseline image creation, switching between multiple baseline image sets,and stitching, can be automated. These features are equally useful in animage tracking context.

As described above, the systems and methods correlate or synchronize thepreviously obtained images with the live images to ensure that anaccurate view of the surgical site, anatomy and hardware, is presentedto the surgeon. In an optimum case, the previously obtained images arefrom the particular patient and are obtained near in time to thesurgical procedure. However, in some cases no such prior image isavailable. In such cases, the “previously obtained image” can beextracted from a database of CT and DRR images. The anatomy of mostpatients is relatively uniform depending on the height and stature ofthe patient. From a large database of images there is a high likelihoodthat a prior image or images of a patient having substantially similaranatomy can be obtained. The image or images can be correlated to thecurrent imaging device location and view, via software implemented bythe image processing device 122, to determine if the prior image issufficiently close to the anatomy of the present patient to reliablyserve as the “previously obtained image” to be interspliced with thelive images.

The display in FIG. 10 is indicative of the type of display and userinterface that may be incorporated into the image processing device 122,user interface 125 and display device 126. For instance, the displaydevice may include the two displays 122, 123 with “radio” buttons oricons around the perimeter of the display. The icons may be touch screenbuttons to activate the particular feature, such as the “label”, “grid”and “trajectory” features shown in the display. Activating a touchscreen or radio button can access a different screen or pull down menuthat can be used by the surgeon to conduct the particular activity. Forinstance, activating the “label” button may access a pull down menu withthe labels “L1”, “L2”, etc., and a drag and drop feature that allows thesurgeon to place the labels at a desire location on the image. The sameprocess may be used for placing the grid and trajectory arrows shown inFIG. 10.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, the same should be considered asillustrative and not restrictive in character. It is understood thatonly the preferred embodiments have been presented and that all changes,modifications and further applications that come within the spirit ofthe invention are desired to be protected.

What is claimed is:
 1. A method for generating a display of an image ofa patient's internal anatomy in a surgical field during a medicalprocedure, comprising: acquiring a high resolution baseline image of thesurgical field including the patient's internal anatomy in a baselineorientation; digitally manipulating the high resolution baseline imageto produce a baseline image set including representative images of thebaseline image at a plurality of permutations of movements of thebaseline image from the baseline orientation; acquiring a new image ofthe surgical field at a lower resolution; comparing the new image to therepresentative images in the baseline image set and selecting therepresentative image having an acceptable degree of correlation with thenew image; and merging the selected representative image with the newimage and displaying the merged image.
 2. The method of claim 1, whereinthe baseline image is one of a pre-procedure full dose fluoroscopicimage or a CT scan image.
 3. The method of claim 1, wherein the baselineimage is a DRR.
 4. The method of claim 1, wherein the new image is oneof a pulse and/or low dose image.
 5. The method of claim 1, whereinpermutations of movements in the step of digitally manipulating thebaseline image includes 4D movements corresponding to a 2D image.
 6. Themethod of claim 1, wherein permutations of movements in the step ofdigitally manipulating the baseline image includes 6D movementscorresponding to a 3D image.
 7. The method of claim 1, wherein: in thestep of digitally manipulating the high resolution image thepermutations of movements form a predefined grid of image movements; andthe step of comparing the new image to the representative images of thebaseline image set includes comparing overlapping pixels between therepresentative image and the new image.
 8. The method of claim 1,wherein the step of comparing the new image to the representative imagesof the baseline image set includes heuristically selectingrepresentative images for comparison.
 9. The method of claim 1, whereinthe step of comparing the new image to the representative images of thebaseline image set includes: performing a principal component analysis(PCA) on the pixels of the representative images in the baseline imageset to generate one or more PCA vectors; producing a PCA matrix of PCAvectors for each pixel in a representative image; generating a columnvector for each representative image and the new image of pixel data foreach pixel in the image; performing a matrix multiplication of the PCAmatrix and each column vector to generate a new column vector for eachrepresentative image and the new image; obtaining the dot product of thecolumn vector for the new image and the column vector for each of therepresentative image; and selecting a representative image for which thedot product is within a pre-determined threshold.
 10. The method ofclaim 1, in which the medical procedure includes tools, instruments,implants or other objects that block or obscure the internal anatomy inan image of the surgical field, wherein the step of comparing the newimage to the representative images of the baseline image set includesonly comparing portions of the images outside the portions that areblocked or obscured.
 11. The method of claim 10, wherein the location ofthe blocked or obscured portions of the new image are determined bydetermining which pixels have a value outside a pre-determinedthreshold.
 12. The method of claim 1, wherein: the step of digitallymanipulating the high resolution baseline image includes providingparallel images to each representative image in which certain anatomicfeatures are reduced or enhanced; and the step of merging the selectedrepresentative image includes merging and displaying the parallel imageto the selected representative image.
 13. An image processing device forgenerating a display of an image of a patient's internal anatomy duringa medical procedure, comprising: a memory for storing a high resolutionbaseline image of the surgical field including the patient's internalanatomy in a baseline orientation and a new image of the surgical fieldat a low resolution; and a processor configured to; digitally manipulatethe high resolution baseline image to produce a baseline image setincluding representative images of the baseline image at a plurality ofpermutations of movements of the baseline image from the baselineorientation; perform software instructions for comparing the new imageto the representative images in the baseline image set and selecting therepresentative image having an acceptable degree of correlation with thenew image; digitally merging the selected representative image with thenew image; and generating signals for displaying the merged image on adisplay device.
 14. The image processing device of claim 13, whereinpermutations of movements include 4D movements corresponding to a 2Dimage.
 15. The image processing device of claim 13, wherein permutationsof movements include 6D movements corresponding to a 3D image.
 16. Theimage processing device of claim 13, wherein: the processor isconfigured to digitally manipulating the high resolution image such thatthe permutations of movements form a predefined grid of image movements;and the software instructions for comparing the new image to therepresentative images of the baseline image set includes comparingoverlapping pixels between the representative image and the new image.17. The image processing device of claim 13, wherein the softwareinstructions for comparing the new image to the representative images ofthe baseline image set includes: performing a principal componentanalysis (PCA) on the pixels of the representative images in thebaseline image set to generate one or more PCA vectors; producing a PCAmatrix of PCA vectors for each pixel in a representative image;generating a column vector for each representative image and the newimage of pixel data for each pixel in the image; performing a matrixmultiplication of the PCA matrix and each column vector to generate anew column vector for each representative image and the new image;obtaining the dot product of the column vector for the new image and thecolumn vector for each of the representative image; and selecting arepresentative image for which the dot product is within apre-determined threshold.
 18. The image processing device of claim 13,in which the medical procedure includes tools, instruments, implants orother objects that block or obscure the internal anatomy in an image ofthe surgical field, wherein the software instructions for comparing thenew image to the representative images of the baseline image setincludes only comparing portions of the images outside the portions thatare blocked or obscured.
 19. The image processing device of claim 18,wherein the location of the blocked or obscured portions of the newimage are determined by determining which pixels have a value outside apre-determined threshold.
 20. The image processing device of claim 13,further comprising a user interface operable to allow manual adjustmentof the degree of digitally merging the selected representative imagewith the new image.
 21. The image processing device of claim 20,wherein: the user interface is further operable to allow manuallyswitching between a display of one or more of the representative image,the new image and the merged image; and the processor generates signalsfor displaying on a display device according to the user interface. 22.A method for generating a display of an image of a patient's internalanatomy in a surgical field during a medical procedure, comprising:acquiring an image of the surgical field with the imaging device in afirst orientation; displaying the acquired image; moving the imagingdevice, patient, or table from the first orientation; tracking themovement of the imaging device, patient, or table from the firstorientation; moving the displayed image in relation to the trackedmovement prior to acquiring a new image of the surgical field with theimaging device.
 23. The method of claim 22, wherein the step of movingthe displayed image includes compensating for errors in the movement ofthe displayed image generated by the position of the imaging devicerelative to the surgical field.
 24. The method of claim 23, wherein thestep of compensating for errors includes determining the center of massfor the surgical field and adjusting the movement of the displayed imagebased on the position of the imaging device relative to the center ofmass.
 25. The method of claim 22, wherein the step of displaying theacquired image includes overlaying an indicia on the displayed imageindicative of a desired field of view for the new image.
 26. The methodof claim 25, wherein the indicia is displayed in a first state when thedisplayed image is outside the desired field of view and a second statewhen the displayed image is within the desired field of view.
 27. Themethod of claim 26, wherein the imaging device is moved in response tothe state of the indicia.
 28. The method of claim 26, wherein thepatient or table is moved while the imaging device remains stationary toposition the patient with the surgical field within the desired field ofview.
 29. The method of claim 25, wherein the desired field of viewcorresponds to an orientation for stitching multiple new images of thesurgical field.
 30. The method of claim 22, therein the step ofdisplaying the acquired image includes overlaying an indicia on thedisplayed image indicative of the position of the imaging devicerelative to a global coordinate system.
 31. The method of claim 22,further comprising the step of overlaying an indicia indicative of adesired movement of the displayed image.
 32. The method of claim 31,wherein the indicia is a grid overlaid on the displayed image thatremains stationary relative to the displayed image as the displayedimages moves.
 33. The method of claim 32, wherein the indicia includes atrajectory indicator indicative of the direction of view for the newimage that moves with the displayed image.
 34. The method of claim 33,wherein the imaging device is moved until the trajectory indicator isaligned with part of the grid prior to acquiring the new image.
 35. Themethod of claim 22, further comprising the step of overlayingidentifiers corresponding to the anatomic features in the displayedimage that move with the displayed image.
 39. The method of claim 22,further comprising; acquiring the new image; comparing the new image tothe displayed image and adjusting the displayed image to eliminate anydrift between the two images.
 37. The method of claim 22, furthercomprising; acquiring the new image; comparing the new image to thedisplayed image and adjusting the displayed image to stabilize thelocation of the anatomy displayed.
 38. The method of claim 22, furthercomprising: receiving position data from an image guidance system; andcorrelating the displayed image to the position data and adjusting thedisplayed image accordingly.
 39. The method of claim 22, in which themedical procedure is a surgical navigation procedure in which theposition of an object, such as a tool or instrument, is tracked, whereinthe new image includes an image of the tool or instrument; and themethod further comprises: introducing a representation of the object onthe acquired image; after moving the displayed image in relation to thetracked movement, acquiring position data corresponding to the trackedposition of the object and comparing the position data with the positionof the object on the moved image; and recalibrating the moved imagebased on the comparison of the position data with the position of theobject on the moved image.
 40. A method for generating a display of animage of a patient's internal anatomy in a surgical field during amedical procedure, in which the medical procedure includes tools,instruments, implants or other objects that block or obscure theinternal anatomy in an image of the surgical field, comprising:acquiring a high resolution baseline image of anatomy within thesurgical field in a baseline orientation; digitally manipulating thehigh resolution baseline image to produce a baseline image set includingrepresentative images of the baseline image at a plurality ofpermutations of movements of the baseline image from the baselineorientation; acquiring a new image of the surgical field in whichportions of the anatomy are blocked by objects; comparing the new imageto the representative images in the baseline image set to select arepresentative image having an acceptable degree of correlation with thenew image; and displaying the selected representative image to show thesurgical field minimizing, intensifying or eliminating the blockingobjects.
 41. The method of claim 40, wherein the step of comparing thenew image to the representative images of the baseline image setincludes only comparing portions of the images outside the portions thatare blocked or obscured.
 42. The method of claim 41, wherein thelocation of the blocked or obscured portions of the new image aredetermined by determining which pixels have a value outside apre-determined threshold.
 43. The method of claim 40, furthercomprising: receiving position data from an image guidance system; andcorrelating the displayed image to the position data and adjusting thedisplayed image accordingly.
 44. The method of claim 40, wherein: thestep of digitally manipulating the high resolution baseline imageincludes providing parallel images to each representative image in whichcertain anatomic features are reduced or enhanced; and the step ofdisplaying the selected representative image includes displaying theparallel image to the selected representative image.
 45. The method ofclaim 1, wherein the comparing step includes: obtaining an intensityvalue for each pixel of each baseline image of said baseline image set;obtaining an intensity value for each pixel of said new image;generating a scalar function of the intensity values for like positionedpixels in said new image and each of the baseline images, the scalarfunction generating a scalar value; and selecting the baseline imagehaving the largest scalar value as said selected representative image.46. The method of claim 45, wherein a Z score is generated for thescalar function of each of the baseline images in relation to scalarfunctions of all the baseline images and the baseline image having thelargest Z score is selected.
 47. The method of claim 45, wherein: atleast the comparing step is performed in a device having a graphicsprocessing unit (GPU); and the multiplication of pixels for generatingthe scalar function occurs simultaneously in multiple processors of theGPU.
 48. The method of claim 45, wherein the step of generating thescalar function includes: generating the scalar function based ondown-sampled images of the baseline images and the new image in whichthe down-sampled images include fewer than all of the pixels of theoriginal baseline and new images.
 49. The method of claim 48, whereinthe step of selecting the image having the largest scalar valueincludes: selecting the down-sampled baseline image having the largestscalar value; further manipulating the selected down-sampled baselineimage at a plurality of permutations of movements of the selected imageto produce second baseline images in a second baseline image set;generating a second scalar value for like positioned pixels in the newimage and each of the second baseline images in the second baselineimage set; and selecting the second baseline image having the largestscalar value as said selected representative image.
 50. The method ofclaim 1, wherein the step of digitally manipulating the high resolutionbaseline image includes filtering to the baseline image to distinguishbetween anatomical and non-anatomical features in the baseline image.51. The method of claim 50, wherein the filtering includes edgedetection.
 52. The method of claim 50, wherein the filtering includes:applying neighborhood functions in a predetermined neighborhood of eachpixel; and identifying each pixel as anatomic if the result of theneighborhood function is outside a predetermined threshold.
 53. Themethod of claim 50, wherein the filtering includes: applyingneighborhood functions in a predetermined neighborhood of each pixel;and identifying each pixel as non-anatomic if the result of theneighborhood function corresponds to a result in a predetermined lookuptable.
 54. The method of claims 52 and 53, wherein the neighborhoodfunction is selected from one or more of standard deviation, gradientand compounded functions of both standard deviation and gradient. 55.The method of claim 54, wherein the neighborhood function is applied toa neighborhood defined as a grid of a predetermined size centered oneach pixel.
 56. The method of claim 55, wherein the grid is five pixelsby five pixels.
 57. The method of claim 55, wherein the grid is threepixels by three pixels.
 58. The method of claim 50, wherein the step ofdigitally manipulating the high resolution baseline image includes:generating a mask corresponding to the non-anatomical features in theimage; and applying the mask to the baseline image to generate amodified baseline image having the non-anatomic features removed. 59.The image processing device of claim 13, wherein; the processor is agraphics processing unit (GPU) having multiple processors; and thesoftware instructions include performing the comparison between imageson a pixel by pixel basis, with each of the multiple processorssimultaneously performing a comparison of different pixels in theimages.
 60. The method of claim 45 wherein the scalar function isselected from the dot product, the correlation coefficient, meanabsolute error and the mean square error.
 61. The method of claim 45wherein the steps of obtaining intensity values include: the userselecting one or more regions of interest in the new image; and scalingthe pixels in the new and baseline images by their distance from theregion(s) of interest.
 62. The method of claim 1, wherein one or both ofthe baseline image and new image is averaged from several consecutiveimages.
 63. The method of claim 1, wherein: the step of acquiring a highresolution baseline image includes acquiring a plurality of highresolution baseline images of the same anatomy and same baselineorientation over successive times; and the step of digitallymanipulating includes manipulating all of the plurality of baselineimages.
 64. The method of claim 1, wherein: the comparing step includesgenerating an image difference from the differences between therepresentative image and the new image; and the merging step includesselectively overlaying the difference image to one or more of the mergedimage, representative image and new image.
 65. The method of claim 64,wherein the merging step includes enhancing the difference image priorto overlaying.
 66. The method of claim 11, wherein the pre-determinedthreshold is selected from one or more of standard deviation, gradientand compounded functions of both standard deviation and gradient of thepixel intensities.
 67. The method of claim 19, wherein thepre-determined threshold is selected from one or more of standarddeviation, gradient and compounded functions of both standard deviationand gradient of the pixel intensities.
 68. The method of claim 42,wherein the pre-determined threshold is selected from one or more ofstandard deviation, gradient and compounded functions of both standarddeviation and gradient of the pixel intensities.
 69. The method of claim1, wherein the comparing step includes selecting successive groups ofimages having an acceptable degree of correlation to be down-sampled inone or more successive iterations to find a final representative image.70. The method of claim 69, wherein the down-sampled images are imagesof increasing resolution in successively smaller regions of interest inthe images.